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ABSTRACT 


Students learn scientific concepts and mathematical calculations relating to scientific 
principles by repetition and reinforcement. Teachers and instructors cannot practically 
spend the long time required during tutorials to patiently teach students the calculations. 
Usually, teachers assign homework to provide practice to students, hoping that spending 
time on the task and providing appropriate feedback will make students stronger in the 
concepts. However, these assignments do not provide insights into how the students 
approached a problem, or which part of the problem stymied them. To provide fine¬ 
grained details about how students interacted with physics problems, a tutor was designed 
with state of the art software tools provided by Carnegie Mellon University's CTAT 
(Cognitive Tutor Authoring Tools). In this article, the results of deploying the computer 
based tutor and the lessons learned from about 2000 fine-grained instances of student 
interactions with the tutoring software are presented. 
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INTRODUCTION 

Improving education in Science, Technology, Engineering, and Mathematics (STEM) is critical to any 
country interested in maintaining a competitive economic advantage. STEM related occupations and job 
opportunities are growing every year. Degree holders in STEM related fields earn a higher income than their 
non-STEM counterparts. STEM education provides the basic tools required for an individual to succeed and 
flourish in society (Xie, Fang, & Shauman, 2015). Physics, a pivotal field of STEM that generates fundamental 
knowledge for future technological advances, serves as the backbone of STEM. 

Despite a large growth of job opportunities, openings for innovation and entrepreneurship, and very 
good chances of success, the number of students opting to study physics related courses in most countries 
has stagnated. Even though efforts are being made to include women and under-represented minorities in 
physics, we are still a long way away from making physics and physics education equally accessible to 
everyone. 

Although physics is an intuitive and interesting subject for any curious thinker, some misconceptions 
and preconceived notions have been attributed to avoidance of physics. First, students tend to think of 
physics as a bunch of mathematical equations without real world applications. Nothing else could be farther 
from the truth. While it is true that physics uses mathematics, the equations are used purely to describe the 
beauty of the laws of nature in an understandable and succinct form; mathematics is often used as a tool to 
represent relationships between different real world quantities. Second, students believe that learning or 
doing physics is an innate ability that cannot be learned. Obviously, this is untrue. Every popular physicist 
started out just like everybody else. Finally, fear of physics is prompted by its portrayal as a "difficult" subject 
(Mallory, 2004). True, physics can be challenging at times, but through proper learning and appropriate 
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practice, proficiency in physics can be built slowly. 

Studies have shown several methods of improving physics learning among students. Hands-on 
activities performed under the guidance of an instructor have been shown to improve knowledge retention 
and student interest in the subject (Swarat, Ortony, & Revelle, 2012). Even if concepts in physics are narrated 
as a story, efforts can be made to sprinkle the recitation with plenty of anecdotes and interesting accounts 
from scientists. When such suspense and thrill is added, students listen with much more interest and retain 
concepts longer (Hidi, 2001; Hidi & Baird, 1986). Anecdotes rife with characters or personalities that students 
can identify with have been shown to increase student interest in science (Krapp, Hidi, & Renninger, 1992). 

Multimedia has played and continues to play an important role in improving physics education. Many 
physics concepts such as rotational motion and three-dimensional dynamics involve stereoscopic thinking. It 
is not possible to explain these concepts on a two dimensional blackboard. By designing multimedia 
appropriate to the level of the target audience (students), it was shown that effective learning can be 
achieved (Jian-hua & Hong, 2012). Recent advances in multimedia authoring tools such as Easy Java 
Simulations have enabled physics teachers to make their own simulations or modify existing simulations for 
classroom use (Esquembre, 2010). Many examples exist, which demonstrate that such simulations can be 
used in the K-12 or college classrooms to maximize physics learning (Wee & Goh, 2013). Recently, 
asynchronous communication among students have been shown to improve learning (Ferdinand-James, 
2017). Use of social media such as Facebook have also been reported to enhance student learning 
(Ferdinand-James & Foogooa, 2017). 

While hands-on activities and multimedia can help in introducing a new concept to students, long term 
retention can only be achieved by repeated reinforcements of principles, and constantly challenging students 
to attempt increasingly difficult tasks (Clifford, 1990). As they attempt these tasks, students are bound to 
make errors. At this point, fast and accurate feedback is required for them to progress in their learning 
(Higgins, Hartley, & Skelton, 2010). The student input-teacher feedback loop serves to improve two-way 
communication between educators and students. This feedback is very informative because it illustrates the 
learning gap between a given student performance and the desired learning outcome (Askew & Lodge, 2000). 
The timing and appropriateness of feedback is critical. Feedback appropriate to the student level should be 
provided, in the quickest turn-around time possible; even better, it can customized to the student, based on 
the nature of the error made. Ideally, a human teacher will be able to provide appropriate feedback to the 
students. But the cost and effort involved in employing a human tutor, especially in a large college classroom, 
can be prohibitive. Computers offer the second best option. If a computer can provide appropriate feedback 
for improving student conceptual understanding and develop problem-solving skills, it would enhance 
learning. 

To be applicable in a classroom, a computer based tutoring system has to have several requirements 
for success. The user interface is the first step in student interaction with the computer system, so it has to 
be designed very carefully. The background and foreground colors of the interface screen should be selected 
for text clarity. The text font size should also be appropriate. Especially in a physics based session, it is 
important to ensure that all the sub-scripts, super-scripts, and Greek symbols are displayed correctly. Too 
small a font size makes it inconvenient to read the text, whereas too big a font size prevents viewing the 
entire problem on one screen. Secondly, the problem solving strategy to be taught should be clearly 
highlighted in a systematic manner. The cues for every step should appear on the screen. It may be a good 
idea to show a general structure of the problem solving steps on one part of the screen, while the tasks to 
be performed by students are shown on the other part. Third, students should be prompted to complete the 
steps involved in solving a problem, in a specified order. This way, the thought process of students can be 
streamlined, and the time spent on problem solving practice can be efficiently utilized. Finally, when students 
make an error while solving the problem in the specified order, appropriate feedback and hints should be 
provided to put them back on track. For data analysis, every step of interaction between the student and the 
tutoring interface has to be exported to a repository. Data can then be downloaded to find patterns or 
common but unanticipated behavior among different classes of students. 

With all these considerations in mind, and to ensure the tutoring system provides flexibility to the 
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author in designing the interface, Cognitive Tutor Authoring Tools (CTAT), hosted by Carnegie Mellon 
University, was chosen because it is provided free to teachers all over the globe. The CTAT based tutor was 
created using the HTML programming environment, and implemented on Tutorshop, a freely available 
platform for deploying CTAT based tutors, hosted by Carnegie Mellon University. Student-computer 
interaction data was streamed to Dotashop, a big data repository for collecting anonymous student data, 
maintained by Carnegie Mellon University (Koedinger, Baker, Cunningham, & Skogsholm, 2010). 

Objective 

It is generally accepted that students who learn from the classroom with the help of a human tutor 
learn concepts better, especially in physics. This project investigated to what extent a computer based tutor, 
designed by a human teacher, can fulfill the needs of a tutor. The objective was realized with a computer 
based tutor that can provide instant feedback, hints based on student performance, and information on 
student-computer interactions to the teacher. 

METHODOLOGY 

A tutor which helps students in the university physics class solve problems in physics, was created using 
CTAT. The lessons in physics introduced in the first year of the university (number of students: 26) are 
designed to provide an overview of the physical laws that govern the world. Many problem solving 
assessment tools involve providing the students with a real life situation, and asking them to solve for an 
unknown quantity, given some known physical quantities. One part of solving this problem is understanding 
the physics concept, and translating it into mathematical equations. The most important part, however, is to 
solve the mathematical equations to determine physical quantities asked by the problem. So, building a 
mathematical background, or refreshing the knowledge of mathematics is very crucial for first year students 
to solve physics problems. This exercise was designed with that end in mind. Even though the focus of this 
paper is not physics, it is worth mentioning a few points here to maintain continuity. A screenshot of the 
tutor made in this project is shown in Figure 1. 


Rotational Motion Series 1: 


lu tlus exeerase, we will learn to use the 
equations of rotational motion: 

Uf = uJi + a t 
Of = Oi + L di t + t 2 
ujf 2 =Wi 2 +2 a (Of - Oi) 
Of = 0{ + 7}(uJi + ujf) t 

These equations involve quantities related to 
rotational motion - initial angular velocity, 
final angular velocity, angular acceleration, 
tune, uutial angular position, and fmal angular 
position. 

In the panel to the right, you find some 
quantities of rotational motion given to you. 
You are required to calculate and enter the 
unkown quantity 


123456789 
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Hint 


Figure 1: User interface for the tutor described in this paper. The students read instructions 
provided on the left panel, and submit their answers by completing the right panel 


The left panel in the interface shows clear instructions regarding this problem. The equations of motion 
corresponding to rotational motion are shown in the figure. These equations allow the student to calculate 
the variables of rotational motion. As shown in the right panel, three of the variables are provided, and 
students are required to calculate the unknown fourth variable,^. They can do this by using one of the 
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equations provided in the left panel, namely, equation 1. As a student performs a calculation and enters a 
value, the tutor responds to the student input. If the calculated value is correct, the tutor presents the next 
problem. If the answer is incorrect, the tutor responds by providing hints that enable the student to redo the 
calculations, and hopefully enter the correct answer. 

As this tutor was implemented, we anticipated two problems. Suppose the student approximates the 
values obtained in the calculation, the program could indicate a 'wrong', even if the student performed 
calculations correctly. To avoid this problem, a tolerance value was programmed into the tutor such that the 
program accepts correct answer + the tolerance value. In this tutorial exercise, the students were given three 
different problem sets (that is, three different variables to be calculated). Each problem set contained 10 
problems to provide practice to students. 


RESULTS AND DISCUSSION 

The tutor intended to teach calculations involving concepts in 'rotational motion' was implemented in 
the first year university physics classroom. About four months before this tutor was introduced, the students 
started out their physics class by learning motion in one dimension to define basic physics concepts; they 
learned Newton's laws to understand how objects behave in the three dimensional world. After learning 
about the applications and principles of energy conservation, momentum, and force, the students embarked 
on their journey to understand rotational motion. They were provided adequate training and knowledge in 
concepts relating to rotational motion, and were tested on how skillfully they can apply their knowledge to 
solve problems involving rotational motion. The results showing how the students performed in this exercise 
are summarized in the graphs in Figure 2: 
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Figure 2: (a)Time taken by the student to complete a problem step, and 
(b) Percentage of wrong attempts made by a student for the tutored problem. 
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Figure 2 summarizes results of student performance in this exercise. Figure 2 (a) shows the average 
time taken for various students to complete one problem of the exercise. The X-axis shows the time taken 
by the students to solve the problem, in a linear scale of 0 to 45 seconds. The Y axis shows the performance 
of every individual student working with the tutor. As shown here, there is a large diversity in student thought 
process and thinking times. While some students were able to finish the problems quickly (thinking time ~10 
sec), others took as much as ~45 sec. to complete a problem. While looking at this data, one may wonder 
why there is such a large range of thinking times, given that all the students were provided the same 
instruction. A look at figure 2 (b) clarifies things a little bit. Figure 2(b) shows the trajectory taken by students 
from the point they started working on the problem to the finishing point. The green bar shows the 
percentage of 'correct' answers, pink shows the 'incorrect' answers, while yellow shows the percentage of 
questions in which the students asked for a 'hint'. As shown in figure 2(b), some students were able to arrive 
at the 'correct' answer for all questions, without asking for a hint, or answering incorrectly. About 81% (21 
among the 26 students) of the students either asked for a hint, or answered incorrectly before getting to the 
right answer. This goes to show that a majority of students were in need of instruction beyond what was 
provided, or slipped while answering the question, even though they know the right answer. Among this set 
of students, most of them were able to subsequently arrive at the correct answer without hints, presumably 
by reworking the problems. A small subset of them, however, were able to arrive at the right answer only 
with the help of the hint provided. 

0 5 10 15 20 25 30 35 40 45 0 25 50 75 100 125 150 175 


Time (sec.) to arrive at wrong answer 

Figure 3: Time taken by students to answer a question (a) Correctly, and (b) Incorrectly. 

We can get a perspective of the student thought processes by looking at figure 3. Figure 3(a) shows 
the amount of time taken by students (seconds) in arriving at the correct answer, while figure 3(b) shows the 
time taken (seconds) in arriving at the wrong answer. Each of the rows represent a different student, 
accounting for all (N = 26) students participating in the activity. We notice from figure 3(a) that there is a 
wide variety of learners, as indicated by the time duration ranging from ~10 seconds to ~45 seconds. On the 
contrary, we notice that the time taken by the students to arrive at a wrong answer is significantly higher 
(~175 seconds). With careful observation, we see that there are specifically two student outliers, who take 
more than 100 seconds thinking about the questions, and yet arrive at the wrong answer. Among the other 
students, there is a gradual variation in thinking times (arriving at wrong answer), from about 3 to 80 seconds. 
Some data in figure 3(b) are not shown because these students did not produce a wrong answer. They got 
the questions correct at the first attempt. 

We note that students who arrive at the answer too early (less than ~5 sec) or too late (more than ~75 
sec) are most likely to get the answer wrong. This could be because they do not spend the minimum amount 
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of time required to double check their answers, or spend too much time overthinking the problem. Among 
students who got the answer correctly, there are about 3 student outliers, who spend more than 20 seconds 
thinking about the problem. There are two plausible reasons for these outliers. One possibility is that the 
students were being meticulous enough to check and recheck their answers before entering an answer input 
into the tutor. The other possibility is that they might have answered the question wrong one or more times, 
and then arrived at the correct final answer, maybe after taking a hint, after several trials. 

CONCLUSIONS 

In summary, research has often shown that a simple lecture based pedagogical approach yields very 
little student learning. Many different research based approaches such as hands-on learning, inquiry based 
experimentation, and lecture demonstrations have been shown to produce better student learning. When it 
comes to providing practice and drill, often repetition and reinforcement are required to ensure student 
learning. 

As human tutors cannot physically be present for inordinate amounts of time with students, we 
developed a computer based tutor using freely available software created and maintained by Carnegie 
Mellon University. This software, called cognitive tutor authoring tool (CTAT) provides an intuitive and 
flexible way for teachers and educators to custom make a tutor for their class. Our tutor, intended to train 
first year college students on problem solving in physics, provided both telescopic and fine grained 
information on how a set of 26 students interacted with the software. By analyzing student data, we were 
able to see that while most students were fairly comfortable working the problems, a significant number of 
students had initial troubles getting to solve these problems. However, interacting with the tutor, they made 
sufficient progress, and by the end of the session, everyone arrived at correct answers, using useful feedback 
and hints programmed into the tutor. From student data, we were able to infer that students who thought 
about the problem too little or too much ended up with wrong answers, whereas spending a reasonable 
amount of time, double checking to see if their answer was right, pointed students mostly to the correct 
answer. Gathering such useful information from student data serves as helpful feedback to the teacher and 
students, so that future versions of the tutor can be deployed with more specific instruction to provide 
students with a better learning experience. While a human tutor can better respond to student's body 
language and teach based on a student's explanation of a problem, a computer based tutor can be used to 
shape the student's thought process in a cost and time effective manner. It can help the teacher improve the 
tutor iteratively by providing detailed feedback on student performance. 
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